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October  24,  1997 


MEMORANDUM  FOR  DIRECTOR,  DEFENSE  FINANCE  AND  ACCOUNTING 

SERVICE 

SUBJECT:  Report  on  the  Second  User  Acceptance  Test  of  the  Electronic  Document 
Management  System  at  the  Defense  Finance  and  Accounting  Service 
Operating  Location,  Omaha,  Nebraska  (Report  No.  98-013) 


We  are  providing  this  evaluation  report  for  your  information  and  use.  This  report 
provides  the  results  of  our  follow  up  on  the  design  and  development  deficiencies  identified 
in  our  earlier  review  of  the  Electronic  Document  Management  system  at  the  Defense 
Finance  and  Accounting  Service  Omaha  Operating  Location.  The  results  of  the  earlier 
review  were  reported  in  IG,  DoD,  Report  Number  97-050,  "Evaluation  of  Controls  Over 
Workflow  Applications  Selected  for  Electronic  Document  Management,"  December  17, 
1996.  The  initial  evaluation  was  performed  in  response  to  a  September  19,  1995,  request 
from  the  Defense  Finance  and  Accounting  Service  (DFAS)  for  assistance  in  reviewing  the 
design  and  development  of  the  Electronic  Document  Management  vendor  payment 
system.  We  announced  this  followup  evaluation  on  November  18,  1996. 

We  provided  a  draft  of  this  report  on  September  5,  1997.  Because  this  report 
contains  no  findings  or  recommendations,  written  comments  were  not  required,  and  none 
were  received.  Therefore,  we  are  publishing  this  report  in  final  form. 

We  appreciate  the  courtesies  extended  to  the  evaluation  staff.  Questions  on  the 
evaluation  should  be  directed  to  Ms.  Kim  Caprio,  Evaluation  Program  Director  at 
(703)  604-9139  (DSN  664-9139  or  KCaprio@DODIG.OSD.MIL)  or  Mr.  Carl  F.  Zielke, 
Evaluation  Project  Manager,  at  (703)  604-9147  (DSN  664-9147  or 
CZielke@DODIG.OSD.MIL).  See  Appendix  B  for  the  report  distribution.  The 
evaluation  team  members  are  listed  inside  the  back  cover. 
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Second  User  Acceptance  Test  of  the  Electronic  Document 
Management  System  at  the  Defense  Finance  and  Accounting 
Service  Operating  Location  Omaha,  Nebraska 


Executive  Summary 


Introduction.  This  report  follows  up  on  the  design  and  development  deficiencies 
identified  in  our  review  of  the  Electronic  Document  Management  system.  The  initial 
evaluation  was  performed  in  response  to  a  September  19,  1995,  request  from  the  Defense 
Finance  and  Accounting  Service  (DFAS)  for  assistance  in  reviewing  the  design  and 
development  of  the  Electronic  Document  Management  vendor  payment  system.  DFAS 
requested  this  review  to  ensure  that  management  and  system  control  requirements  were 
met  before  system  acceptance. 

The  first  user  acceptance  training  and  test  was  conducted  from  April  1  through 
May  28,  1996.  Because  of  design  and  development  deficiencies  identified  during  the  first 
user  acceptance  test  at  the  DFAS  Omaha  Operating  Location,  testing  was  suspended  on 
May  28,  1996,  so  that  the  contractor  could  develop  and  test  solutions  to  the  identified 
deficiencies.  We  reported  the  results  of  the  first  user  acceptance  test  in  IG,  DoD,  Report 
No.  97-050,  "Evaluation  of  Controls  Over  Workflow  Applications  Selected  for  Electronic 
Document  Management,"  December  17,  1996.  On  June  20,  1996,  the  contractor  provided 
"Solutions  A  -  T  for  the  DFAS  Electronic  Document  Management  Partnership,"  which 
described  the  solutions  and  schedule  for  completion  of  the  deficiencies  reported  during  the 
first  test.  The  second  user  acceptance  test  was  from  November  19  through  December  20, 
1996.  We  announced  this  followup  evaluation  of  that  test  on  November  18,  1996. 

Evaluation  Objectives.  The  overall  evaluation  objective  was  to  determine  whether  the 
Electronic  Document  Management  (EDM)  system  can  satisfactorily  execute  vendor 
pajmient  workflows  while  providing  adequate  security  for  system  and  production  data. 
Specifically,  we  determined  whether  system  performance  and  control  deficiencies 
identified  during  the  first  user  acceptance  test  were  corrected. 

Evaluation  Results.  We  commend  DFAS  for  resolving  the  functional  deficiencies 
reported  in  the  first  user  acceptance  test.  Based  on  the  results  of  the  second  acceptance 
test,  we  believe  the  EDM  system  is  capable  of  performing  in  an  operational  environment 
the  tasks  assigned  for  the  vendor  payment  process.  The  only  open  exception  is 
noncompliance  with  the  trusted  computer  system  criteria  defined  in  DoD  5200.28-STD, 
“Department  of  Defense  Trusted  Computer  System  Evaluation  Criteria,”  December  1985. 
The  noncompliance  is  due  to  an  incompatibility  between  the  EDM  workflow  application 
software  and  the  DFAS  server’s  UNIX-based  operating  system,  which  DFAS  and 
Electronic  Data  Systems,  the  EDM  contractor,  are  working  to  correct.  As  a  result,  the 
EDM  system  was  determined  ready  for  the  next  milestone  in  its  development,  an 


independent  test  by  the  Joint  Interoperability  Test  Command.  The  results  of 
theindependent  test  were  favorable.  For  further  discussion  of  the  evaluation  results,  see 
Part  I. 


Management  Comments.  We  provided  a  draft  of  this  report  on  September  5,  1997. 
Because  this  report  contains  no  findings  or  recommendations,  written  comments  were  not 
required,  and  none  were  received.  Therefore,  we  are  publishing  this  report  in  final  form. 
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Evaluation  Background 


In  1995,  DFAS  selected  the  Omaha  Operating  Location  (OPLOC)  for  the  design, 
development,  and  implementation  of  the  Electronic  Document  Management  (EDM) 
system  prototype  for  vendor  payments.  On  September  19,  1995,  the  Defense  Finance 
and  Accounting  Service  (DFAS)  requested  that  the  Office  of  the  Inspector  General, 
DoD,  review  the  design  and  development  of  the  EDM  system  to  ensure  that 
management  and  system  control  requirements  were  met  before  DFAS  accepted  the 
system. 

The  first  EDM  system  user  acceptance  training  and  testing  was  performed  at  the  Omaha 
OPLOC  from  April  1  through  May  28,  1996.  The  primary  purpose  of  user  acceptance 
testing  is  to  determine  whether  the  EDM  system  could  operate  in  a  production 
environment  for  the  Omaha  OPLOC.  Both  functional  and  technical  areas  were  tested 
and  evaluated.  The  test  was  used  as  the  basis  for  determining  partial  contractual 
acceptance  of  the  overall  EDM  solution,  as  designed,  developed  and  delivered  by  the 
contractor.  Electronic  Data  Systems  (EDS). 

On  May  23,  1996,  the  EDM  Project  manager  at  the  DFAS  Omaha  OPLOC  briefed  the 
DFAS  Configuration  Control  Board,  recommending  that  testing  be  suspended  because 
21  functional  and  9  technical  requirements  were  identified  needing  system-wide 
solutions  before  the  EDM  system  acceptance  test  could  resume.  On  June  7,  1996, 
DFAS  and  EDS  formally  detailed  the  system  problems  and  requirements,  and  on 
June  20,  1996,  EDS  submitted  “Solutions  A-T  for  the  DFAS  Electronic  Document 
Management  Partnership,”  which  documented  proposed  solutions  and  a  schedule  for 
completion.  Based  on  approval  of  the  solutions  to  those  deficiencies  reported  in  the 
first  user  acceptance  test,  DFAS  scheduled  the  second  user  acceptance  test  from 
November  19  through  December  20,  1996. 

On  December  17,  1996,  we  issued  Report  No.  97-050,  “Evaluation  of  Controls  Over 
Workflow  Applications  Selected  for  Electronic  Document  Management,”  which 
identified  4  functional  and  3  technical  deficiencies  requiring  improvement.  The 
functional  deficiencies  identified  included  document  capture  and  indexing,  accounts 
payable  workflows,  voucher  certification  workflow,  and  disbursing  and  for-others 
workflows.  The  technical  deficiencies  identified  were  unlimited  attempts  to  log  on  to 
the  local  area  network  (LAN),  the  audit  trail  application  not  being  activated,  and 
inadequate  secure  file  protection  that  does  not  meet  Controlled  Access  Protection  under 
the  DoD  5200.28-STD,  “Department  of  Defense  Trusted  Computer  System  Evaluation 
Criteria,”  December  1985. 

On  December  20,  1996,  DFAS  determined  that  the  EDM  solutions  and  full  system 
were  functionally  and  technically  tested  in  sufficient  detail  to  fully  demonstrate  all 
aspects  of  the  design  associated  with  the  electronic  capture,  routing,  and  pro^ssmg  of 
documents.  At  the  conclusion  of  the  test  period,  DFAS  determined  that  the  EDM 
solution  was  ready  for  transition  from  the  test  environment  to  an  operational  production 
environment  (Milestone  III).  This  report  discusses  the  actions  taken  by  DFAS  to 
address  the  four  functional  and  three  technical  deficiencies  that  we  addressed  m  our 
December  17,  1996,  audit  report. 
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Evaluation  Objectives 


The  overall  evaluation  objective  was  to  determine  whether  the  EDM  system  can 
satisfactorily  execute  vendor  payment  workflows  while  providing  adequate  security  for 
system  and  production  data.  Specifically,  we  determined  whether  system  performance 
and  control  deficiencies  identified  during  the  initial  acceptance  test  period  were 
corrected.  For  a  discussion  of  audit  scope  and  methodology  and  prior  coverage  related 
to  the  evaluation  objectives,  see  Appendix  A. 
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Second  User  Acceptance  Test 

Based  on  the  results  of  the  second  user  acceptance  test,  the  EDM  system 
has  demonstrated  that  it  can  perform  in  an  operational  environment  the 
tasks  assigned  for  the  vendor  payment  process.  Functional  and  technical 
weaknesses  reported  earlier  have  been  corrected.  The  only  open  exception 
is  trusted  system  compliance  due  to  a  vendor  limitation  that  DFAS  and 
EDS  are  working  to  correct.  Although  important,  the  exception  was  not 
sufficient  to  delay  testing  of  the  system.  In  addition,  DF  AS  implemented 
an  effective  user  testing  process  that  ensured  user  participation  and 
developer  support.  As  a  result,  the  EDM  system  received  a  favorable 
report  on  an  independent  test  by  the  Joint  Interoperability  Test  Command. 


Test  Purpose 


The  primary  purpose  of  user  acceptance  testing  is  to  determine  whether  the 
EDM  system  could  operate  successfully  in  a  production  environment  to  meet  the 
user's  needs.  The  first  user  acceptance  training  and  test  conducted  from  April  1 
through  May  28,  1996,  resulted  in  30  functional  and  technical  deficiencies  that 
needed  system-wide  solutions  before  the  EDM  system  could  be  accepted  by 
DFAS.  The  second  user  acceptance  test  was  conducted  to  determine  whether 
the  developed  solutions  corrected  the  identified  deficiencies  and  prepared  the 
EDM  system  for  Initial  Operational  Test  and  Evaluation  (lOT&E). 

The  second  user  acceptance  test  followed  a  structured  approach  to  evaluate  the 
solutions  and  all  aspects  of  the  EDM  system.  The  test  included  a  designated 
team  of  51  testers  from  varied  locations:  DFAS  Headquarters,  DFAS  Denver 
Center,  DFAS  Omaha  OPLOC,  DFAS  Indianapolis  Center,  DFAS  Orlando 
OPLOC,  DFAS  Charleston  OPLOC,  DFAS  Columbus  Center,  and  DFAS 
Kansas  City  Center.  Testers  followed  specific  instructions  on  134  test  scenanos 
to  support  both  functional  and  technical  evaluations  and  documented  the  results. 
The  testers  generated  228  problem  reports  that  were  analyzed  to  determine  ways 
for  correction.  Only  one  problem  report  relating  to  user  authentication 
remained  after  the  test  which  DFAS  is  working  with  the  developer  to  correct. 

As  a  result  of  the  successful  test,  DFAS  determined  that  the  EDM  system  was 
ready  for  independent  testing. 

The  Joint  Interoperability  Test  Command  (JTIC),  Fort  Huachuca,  Arizona,  the 
operational  test  agency  for  the  DFAS,  conducted  the  EDM  lOT&E  from 
May  27  through  June  6,  1997.  The  primary  purpose  of  the  lOT&E  is  to 
determine  whether  the  EDM  system  is  operationally  effective  and  suitable  for 
the  intended  users.  The  results  of  the  lOT&E  are  provided  to  the  Milestone 
Decision  Authority  and  the  Program  Manager  in  an  Independent  Evaluation 
Report  that  is  used  to  support  requirements  for  the  Milestone  III  decision  to 
deploy  the  system.  The  lOT&E  results  were  favorable. 
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Test  Results 


DFAS  should  be  commended  for  resolving  the  functional  and  technical  issues 
reported  in  their  first  user  acceptance  test.  The  EDM  system  at  the  Omaha 
OPLOC  has  demonstrated  that  it  can  perform  in  an  operational  environment  the 
tasks  assigned  for  the  vendor  payment  process.  The  only  exception  is 
noncompliance  with  DoD  trusted  computer  system  criteria  as  defined  in 
DoD  5200.28-STD.  Trusted  computer  systems  are  able  to  simultaneously 
process  a  range  of  sensitive  unclassified  or  classified  information  for  a  diverse 
set  of  users  without  violating  access  privileges.  The  noncompliance  is  due  to  an 
incompatibility  with  the  EDM  workflow  application  software  and  the  DFAS 
server’s  UNIX-based  operating  system,  which  affects  secure  file  protection  on 
the  UNIX  server.  DFAS  and  EDS  are  working  to  correct  this  deficiency.  The 
second  user  acceptance  test  verified  that  the  proposed  solutions  to  the  functional 
and  technical  deficiencies  reported  during  our  first  evaluation,  work  as  intended. 
DFAS  implemented  an  effective  user  testing  process  that  ensured  user 
participation  and  developer  support. 

Identified  Functional  Weaknesses.  Four  functional  control  weaknesses  were 
identified  during  the  first  user  acceptance  test:  document  capture  and  indexing, 
accounts  payable  workflows,  voucher  certification  workflow,  and  disbursing 
and  for-others  workflows.  During  the  second  user  acceptance  test,  these  control 
weaknesses  were  corrected,  retested,  and  determined  adequate. 

Document  Capture  and  Indexing.  Document  capture  personnel  misrouted 
incoming  fax  documents;  in  addition,  multiple-page  incoming  fax  documents 
were  split  by  personnel  unfamiliar  with  vendor  payment  documents.  As  a 
result,  documents  were  forwarded  to  the  wrong  vendor  pay  indexing  area  or 
were  inappropriately  split  or  missing.  As  a  part  of  the  second  user  acceptance 
test,  procedures  were  modified  for  incoming  electronic  fax  documents  to  be 
automatically  routed  to  the  indexer  (tier  2)  screens  instead  of  the  document 
capture  center.  This  allows  indexing  personnel,  who  are  more  knowledgeable 
of  the  vendor  pay  process,  to  split  the  documents  appropriately  and  route  those 
documents  to  the  correct  workflow  more  quickly. 

Accounts  Payable  Workflows.  Accounts  payable  workflows  required 
technicians  to  return  documents  to  the  document  capture  center  for  rescanning 
or  exception  handling  when  document  types  needed  to  be  changed,  workflow 
cases  needed  to  be  canceled,  or  new  cases  needed  to  be  created.  Once  the 
document  was  deleted,  rescanned,  and  sent  through  the  indexing  process,  a  new 
workflow  case  was  created  for  that  re-indexed  document.  The  tested  solution 
showed  that  workflow  software  was  modified  to  allow  the  accounts  payable 
team  leader  to  change  document  types,  cancel  old  workflow  cases,  and  create  a 
new  case  when  documents  were  indexed  incorrectly.  Productivity  increased 
because  technicians  could  forward  documents  directly  to  the  team  leader  for 
correction  instead  of  waiting  for  the  document  capture  center  to  locate  the 
original  document,  delete  the  image,  and  rescan  as  appropriate.  The  time  spent 
in  returning  documents  to  the  document  capture  center  can  now  be  spent  more 
productively  on  current  workload. 

Voucher  Certification  Workflow.  The  voucher  certification  workflow  did  not 
allow  the  voucher  to  be  voided  after  certification.  The  EDM  system  uses  the 


5 


Second  User  Acceptance  Test 


Integrated  Accounts  Payable  System  voucher  print  file  by  converting  the 
voucher  to  an  image  and  creating  a  workflow  case  for  each  voucher.  This 
voucher  workflow  case  is  inserted  into  the  workflow  for  certification.  During 
this  process,  supporting  document  images  (such  as  invoices  and  receiving 
reports)  are  added  to  each  workflow  case  by  linking  contract  numbers,  invoice 
numbers,  and  dates  when  goods  and  services  are  received.  The  certifying 
officers  view  the  workflow  cases  that  contain  the  voucher  and  supporting 
documents  images  and  certify,  void,  or  return  the  voucher  assembly  to  an 
accounts  payable  technician  for  additional  review.  The  EDM  system  could  not 
void  vouchers  after  certification,  even  those  certified  in  error.  The  second  user 
acceptance  test  showed  that  workflow  software  had  been  modified  to  allow  for 
voiding  a  certified  voucher  until  the  check  print  file  is  sent  to  the  DFAS  Denver 
Center,  Denver,  Colorado,  for  payment  processing.  This  modification  saves 
time  when  the  automatic  assembly  of  supporting  documentation  results  in  an 
error  and  allows  the  certifying  officer  to  void  a  previously  certified  voucher 
when  incorrect  supporting  documentation  is  discovered. 

Disbursing  and  For-Others  Workflows.  Disbursing  personnel  had  difficulty 
reconciling  the  check  print  file  sent  to  the  DFAS  Denver  Center  because  the 
types  of  payments  were  not  listed.  The  DFAS  Omaha  OPLOC  disbursing 
personnel  reconcile  the  check  print  file  to  ensure  that  the  EDM  records  agree 
with  the  Integrated  Accounts  Payable  System  payment  file.  The  workflow  for 
forwarding  automated  payments  to  the  DFAS  Denver  Center  for  payment 
processing  required  correction  to  show  separate  totals  for  cash,  check,  and 
electronic  funds  transfer  payments  in  the  reconciliation  report.  No 
modifications  were  needed  to  the  for-others  workflow.  The  disbursing 
workflow  for  automated  payments  was  modified  to  show  the  various  types  of 
payments  and  make  voucher  status  data  available  for  the  disbursing  personnel  to 
reconcile  voided  and  certified  vouchers  between  the  Integrated  Accounts 
Payable  System  and  EDM.  Now  the  disbursing  office  electronically  forwards 
the  reconciled  and  certified  check  print  files  listing  the  various  types  of 
payments  to  the  DFAS  Denver  Center  for  payment  processing,  which  in  turn 
enables  payments  to  be  made  promptly. 

Identified  Technical  Weaknesses.  Three  technical  control  weaknesses  were 
identified  during  the  first  EDM  system  acceptance  test.  The  results  of  the  first 
test  showed  that  the  system  could  not  demonstrate  the  necessary  log-on  security , 
audit  trails,  and  safeguards  for  protecting  secure  files.  During  the  second  user 
acceptance  test,  the  procedures  for  logging  on  to  the  local  area  network  (LAN) 
were  corrected  and  audit  trail  software  was  activated.  Requirements  for  secure 
file  protection  were  not  implemented  due  to  a  vendor  limitation  that  DFAS  and 
EDS  are  working  to  correct. 

Loe-on  Security.  The  DFAS  Omaha  OPLOC  operating  system  allowed 
unlimited  attempts  to  log  on  to  the  LAN.  However,  the  system  administrator 
could  determine  the  number  of  attempts  allowed  to  minimize  the  potential  of 
misuse  or  fraud.  For  the  second  user  acceptance  test,  the  system  administrator 
modified  the  operating  system  to  allow  for  only  three  log-on  attempts  before  the 
user  was  blocked  from  further  attempts.  Accordingly,  after  the  third  attempt, 
the  user  must  notify  the  system  administrator  to  reset  the  operating  system. 
System  control  is  enhanced  because  it  alerts  the  system  administrator  to 
unauthorized  attempts  to  enter  the  system  and  also  complies  with  Federal 
Information  Processing  Standard  Publication  112,  "Password  Usage,  May  30, 
1985. 
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Audit  Trails.  Features  of  the  off-the-shelf  software  designed  to  ensure 
adequate  audit  trails  were  not  activated  during  the  first  user  acceptance  test. 

Data  must  be  available  for  reconstruction  of  any  user  session  to  aid  security 
review  or  audit.  During  the  second  user  acceptance  test,  off-the-shelf  audit  trail 
software  was  activated  for  specific  test  scenarios.  The  results  showed  that 
adequate  information  is  being  collected  and  that  system  recovery  is  adequate  to 
ensure  that  data  would  not  be  lost. 

Secure  File  Protection.  The  secure  file  protection  option  on  the  DFAS  server’s 
UNIX-based  operating  system  was  not  implemented  in  order  to  maintain 
compatibility  with  the  EDM  workflow  application  software.^  Therefore,  the 
password  file  in  Unix  is  not  hidden  from  potential  unauthorized  system  users. 
Once  access  to  Unix  server  is  obtained,  individuals  can  read  the  list  of  user 
identification  codes  in  the  secure  password  file  and  attempt  to  decode  the 
encrypted  passwords,  including  the  system  administrator  password.  Once  the 
system  administrator  password  is  obtained,  an  unauthorized  user  could  sltcr  any 
file  on  the  EDM  server,  including  the  audit  files,  thus  eliminating  any  evidence 
of  intrusion. 

The  results  of  the  first  and  second  EDM  system  acceptance  tests  showed  that  the 
system  did  not  demonstrate  controlled  access  protection.  According  to 
DoD  5200.28-STD,  Class  2  controlled  access  protection  (C2)  enforces  more 
discretionary  access  control  by  making  users  individually  accountable  for  their 
actions  through  log-on  procedures  and  auditing  of  security-relevant  events. 


Anoendix  I  to  the  "Defense  Finance  and  Accounting  Service  Electromc 
Document  Management  (EDM)  Program  Increment  1  -  Vendor  Pay  Operational 
Requirements  Document,"  April  3,  1997,  states  that  because  the  EDM  system  is 
based  on  the  Wang  Open/Image  product  requiring  the  Network  Information 
Services  application,  the  Hewlett-Packard  secure  (trusted)  computer  system 
cannot  be  implemented.  Therefore,  the  EDM  system  is  not  C2  compliant.  The 
developer  states  that  noncompliance  results  because  the  trusted  system  is  not 
compatible  with  the  Network  Information  Service  configuration  used  by  EDM 
to  provide  basic  user  authentication  for  the  Wang  products. 

The  DFAS  Omaha  OPLOC  personnel  are  working  to  make  the  EDM  systern  C2 
compliant.  According  to  the  “Test  and  Evaluation  Master  Plan  for  Electronic 
Document  Management  Program,”  May  23,  1997,  the  DFAS  Omaha  OPLOC 
has  implemented  minimum  security  requirements  to  include  the  use  ot  user- 
identifications  and  passwords:  anti-virus  software:  controlled  access  to  the 
LAN’  user  security  clearances:  user  accountability:  security  training  and 
awareness:  and  physical  security  controls.  During  the  second  user  acceptance 
test  we  noted  security  awareness  and  controlled  access  to  certain  functionalities 
based  on  role  definition  of  the  users  and  physical  security  controls  for  the  image 
control  room.  DFAS  Headquarters  has  decided  to  transition  to  a  Windows  N 1 
platform  throughout  the  agency  beginning  in  FY  1998  that  will  support  C2 
requirements.  However,  at  the  time  of  our  evaluation,  neither  Unix  or  the 
Wang  software  were  C2  compliant. 
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Second  User  Acceptance  Test 


Summary 


DFAS  implemented  an  effective  user  testing  process  that  ensured  user 
participation  and  developer  support.  For  the  second  user  acceptance  test,  EDS 
provided  system  support  for  the  solutions  to  the  30  system-wide  deficiencies 
resulting  from  the  first  user  acceptance  test.  Using  134  specifically  developed 
test  scenarios,  51  users  tested  those  solutions  both  functionally  and  technically 
and  determined  that  the  EDM  system  can  perform  in  an  operational  environment 
the  tasks  assigned  for  the  vendor  payment  process  at  the  Omaha  OPLOC.  The 
only  exception  is  trusted  system  compliance  due  to  a  vendor  limitation  that 
affects  secure  file  protection  that  DFAS  is  working  to  correct.  As  a  result,  the 
EDM  system  participated  in  the  Initial  Operational  Test  and  Evaluation  from 
May  27  through  June  6,  1997.  The  results  of  that  test  were  favorable  and 
support  requirements  for  the  Milestone  III  decision  to  deploy  the  system.  For 
these  reasons,  we  are  making  no  further  recommendations. 
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Part  n  -  Additional  Information 


Appendix  A.  Evaluation  Process 


Scope  and  Methodology 


Work  Performed.  For  fiscal  years  1995  through  2002,  Electronic  Document 
Management  (EDM)  system  program  life  cycle  costs  are  estimated  at  $386.8 
million.  We  participated  in  the  second  user  acceptance  test  from  November  19 
through  December  20,  1996.  The  scope  of  this  evaluation  included  the  testing 
of  the  solutions  provided  by  EDS  in  "Solutions  A  -  T  for  the  DFAS  Electronic 
Document  Management  Partnership,"  June  20,  1996.  That  document  is  the 
outcome  of  the  first  user  acceptance  test  that  resulted  in  30  system-wide 
deficiencies. 

Use  of  Computer-Processed  Data.  Computer-processed  data  supporting  the 
second  user  acceptance  test  are  determined  to  be  reliable.  To  achieve  the 
evaluation  objective,  we  extensively  relied  on  computer-processed  data 
contained  in  the  Oracle  database  for  the  EDM  system.  We  assessed  the 
reliability  of  these  data  through  the  user  acceptance  test  scenarios  and  found 
them  to  be  adequate.  As  a  result  of  the  tests  and  assessments,  we  conclude  that 
the  computer-processed  data  are  sufficiently  reliable  for  meeting  the  evaluation 
objectives. 

Evaluation  Type,  Dates,  and  Standards.  We  performed  this  program 

evaluation  from  November  1996  through  July  1997  in  accordance  with 

standards  implemented  by  the  Inspector  General,  DoD.  Our  scope  was  limited 
in  that  we  did  not  include  tests  of  management  controls. 

Contacts  During  the  Evaluation.  We  visited  or  contacted  individuals  and 
organizations  within  DoD.  Further  details  are  available  on  request. 


Summary  of  Prior  Coverage 


IG.  DoD,  Report  No.  97-050,  "Evaluation  of  Controls  Over  Workflow 
Amilications  Selected  for  Electronic  Document  Management,"  December  17, 
1996,  reported  the  results  of  the  first  user  test.  The  overall  objective  of  the 
evaluation  was  to  determine  whether  the  EDM  system  could  achieve 
management  control  objectives  related  to  the  completeness,  accuracy,  and 
authorization  of  data  and  whether  the  system  could  meet  requirements  for 
document  retention.  Specifically,  we  determined  whether  controls  over 
workflow  applications  selected  for  EDM  were  adequate.  The  first  EDM  sy stern 
acceptance  test  showed  that  controls  over  the  EDM  vendor  payment  process  and 
workflows  can  achieve  management  control  objectives  related  to  the 
completeness,  accuracy,  and  authorization  of  data.  However,  improvements 
were  needed  in  the  security  controls  over  EDM  system  data.  Specifically, 
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Appendix  A.  Evaluation  Process 


controls  were  needed  to  limit  log-on  attempts,  meet  auditability  requirements, 
and  protect  secure  files.  DFAS  initiated  corrective  action;  therefore,  no 
recommendations  were  made  in  the  report. 


Appendix  B.  Report  Distribution 


Office  of  the  Secretary  of  Defense 

Under  Secretary  of  Defense  (Comptroller) 

Deputy  Chief  Financial  Officer 
Deputy  Comptroller  (Program/Budget) 

Deputy  Under  Secretary  of  Defense  (Acquisition  Reform) 
Assistant  Secretary  of  Defense  (Public  Affairs) 

Director,  Defense  Logistics  Studies  Information  Exchange 


Department  of  the  Army 

Auditor  General,  Department  of  the  Army 


Department  of  the  Navy 

Assistant  Secretary  of  the  Navy  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Navy 


Department  of  the  Air  Force 

Assistant  Secretary  of  the  Air  Force  (Financial  Management  and  Comptroller) 
Auditor  General,  Department  of  the  Air  Force 


Other  Defense  Organizations 

Director,  Defense  Contract  Audit  Agency 
Director,  Defense  Finance  and  Accounting  Service 
Director,  Defense  Logistics  Agency 
Director,  National  Security  Agency 

Inspector  General,  National  Security  Agency 
Inspector  General,  Defense  Intelligence  Agency 

Non-Defense  Federal  Organizations  and  Individuals 

Office  of  Management  and  Budget  .  ,  .«>•  •  • 

Technical  Information  Center,  National  Security  and  International  Affairs  Division, 

General  Accounting  Office 
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Chairman  and  ranking  minority  member  of  each  of  the  following  congressional 
committees  and  subcommittees: 

Senate  Committee  on  Appropriations 

Senate  Subcommittee  on  Defense,  Committee  on  Appropriations 
Senate  Committee  on  Armed  Services 
Senate  Committee  on  Governmental  Affairs 

House  Committee  on  Appropriations  .  . 

House  Subcommittee  on  National  Security,  Committee  on  Appropriations 

House  Committee  on  Government  Reform  and  Oversight 

House  Subcommittee  on  Government  Management,  Information,  and  Technology, 

Committee  on  Government  Reform  and  Oversight  .... 

House  Subcommittee  on  National  Security,  International  Affairs,  and  Criminal  Justice, 
Committee  on  Government  Reform  and  Oversight 
House  Committee  on  National  Security 
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